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ON LEAST SQUARES FITTING NONLINEAR SUBMODELS 



Gordon G. Bechtel 
Oregon Research Institute 

Abstract 

Three simplifying conditions are given for obtaining least squares 
(LS) estimates for a nonlinear submodel of a linear model. If these are 
satisfied, and if the subset of nonlinear parameters may be LS fit to the 
corresponding LS estimates of the linear model, then one attains the 
desired LS estimates for the entire submodel. Two illustrative analyses 
employing this method are given, each involving an Ecka r t-Young (LS) decom- 
position of a matrix of linear LS estimates. In each case the factors provide 
an LS fit of the nonlinear submodel to the original data. The minimum error 
sum of squares for this fit is the error sum of squares for the corresponding 
linear model plus a function of the eigenvalues involved in the factoriza^ 
tion. An Ecka r t-Young factorization, however, is only a special case of an 
LS decomposition of LS estimates. The present method is more generally 
applicable (under the three simplifying conditions) whenever any LS procedure 
may be found for fitting certain parameters of a nonlinear submodel to the 
corresponding LS estimates of a linear model. 
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ON LEAST SQUARES FITTING NONLINEAR SUBMODELS 
Gordon G. Bechtel 
Oregon Research Institute 

Least squares estimation procedures are readily available for linear 
models which, depending upon their structure, generate "regression analyses" 
or "variance analyses" for particular data layouts. Since the parameters 
of nonlinear models are more difficult to estimate in an analytic way, 
their estimates are usually obtained by approximate iterative techniques. 
However, these techniques suffer from problems of local minima and may 
be rather unwieldy. Therefore, it is the purpose of this paper to 
present conditions which, if met by a particular nonlinear model, simplify 
the analytic problem of finding exact least squares estimates for its 
pa rame ters . 

Our approach is similar to that of the analysis of variance, where 
constraints (hypotheses) generate a 1 i nea r submode 1 of a linear model. 

However, we shall use the general linear model as a device for embracing 
a nonJ_inea_r submodel of interest. This device makes it possible, under 
certain conditions, to obtain exact LS estimates for all of the submodel 
parameters by (l) LS fitting only the nonlinear parameters to the cor respond i ng 
LS estimates of the linear model, and (2) preserving the remaining LS 
estimates of the linear model. In this way some difficult nonlinear 
least squares problems may be reduced to manageable form. Moreover, 
when the major interest is in the linear model itself, this technique consti- 
tutes an adjunct analysis, i.e., a further data breakdown, for the analysis 
of variance associated with that linear model . 

The theoretical part of the paper oresents the general linear model, 
and its nonlinear submodel, in partitioned form. This partitioning identifies 
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the subset of IS estimates to be submitted to further LS decomposition. 

It also oermits the error sum of squares for the nonlinear submodel to 
be written as the (fixed and known) minimum error sum of squares for 
the linear model p 1 us the sum of two other partitioned terms. These 
two terms represent additional error incurred under the nonlinear hypothe- 
sis, and this excess error sum of squares is analogous to the (linear) 
hypothesis sum of squares in the analysis of variance. 

Subsequently, three conditions are invoked to reduce the sum of these 
two partitioned terms to a multiple of another error sum of squares, 
i.e., that for fitting the nonlinear parameters to the corresponding LS 
estimates of the linear model. The analytic minimization of this latter 
error sum of squares, if available, minimizes the entire error sum of 
squares for fitting the nonlinear submodel to the data. Two specific 
nonlinear problems, each satisfying these three conditions, and each 
amenable to an absolute minimization of the second error sum of squares, 
illustrate the usefulness of the method. 



Conceptual Approach 

The Linear Model 

Since the conceptual approach rests upon pa rt i t i oned. vectors and 
matrices, we write the general linear model as 



(D 



y = (X>Z) + e 

1 \VJ 



where ( X ! Z ) is the design matrix and is the parametric vector. The 

i W 

mode! is fit to the vector y of observations with a resulting error vector 



e. A vector (---) of least squares estimates is one which minimizes the 
error sum of squares e^e at e"e, where 
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T he Nonlinear Submodel 

The nonlinear model of interest may be generated by placing a 
hypothes i s 

(2) H : y = g(a) 

upon the linear model. The function g is a nonlinear constraint upon 
y in the argument a, which is some specified set of parameters. We 
then write the nonlinear submodel, i.e., the conjunction of (l) and (2), 
as 




where f is a vector of errors. Our purpose in generating (3) as a submodel 
of ( 1 ) is not to test hypothesis (2), as is usually the case, but 
rather to use (2) as a device for LS estima tion. This will be accomplished 
through the manipulation of the error sum of squares f'f for (3). 

The Error Sum of Squares for the N onli near Submodel 

We may write the error vector for the submodel as 




where e=y- g(a). The least squares problem for (3) is that of choosing 
3 , a so as to minimize the error sum of squares 

(4) ^(e,e) = f'f 
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Equation (4) shows thatj/ 3 is the sum of a known, fixed value e"e, which 
is the error sum of squares for the corresponding linear model, and 
two other terms, which depend upon the submodel parameters 6, a . 

LS Fitting a to y 

The following three paragraphs give conditions for obtaining an 
LS fit of £,a to y by means of an LS f i t of a to y: 

Submodel invariance of B - The elements of P enter (3) 1 i near 1 y , 

vjh i 1 e those of a enter this submodel non 1 t near ly . In certain applications 
it may be possible to find the LS estimate B under the submodel, i.e., 
the value of B which minimizes , wi thout simultaneously solving for 
the least squares estimate a* Moreover, if 

(s) e = § , 

which is the LS estimate under the corresponding linear model, then 
this value may be inserted into (4) reducing ^(p,e) to 

(6) (e) = e'e + 2e'Ze + e'Z'Zz . 

<f' i s the sum of the error sum of squares for the corresponding linear 
model, a bilinear form in 2, and a quadratic form in Z'Z. Since these 
forms depend only upon e, and hence only upon a, the LS solution for 
the nonlinear submodel is completed by choosing a to minimize the sum 
of these bilinear and quadratic forms. Of course, this itself may pre- 
sent a formidable problem, and, therefore, we invoke two more conditions 
to reduce the latter sum to a multiple of e'e . 

Vanishing bilinear form e'le . If in a particular analysis the 
structure of e, Z, and e Is such that 

(7) e 'Ze = 0, 
then we have 
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(8) O^'U) = e'e + z'l'lz , 

which depends only upon a through the quadratic form in I'l. 

Reducible quadratic form e "Z "Ze . The structure of Z and e may 
also permit the reduction 

(9) z'Z'Zz = ce'e , 

of the quadratic form, where c is a known constant characteristic of the 
particular analysis. In this case, we can write 

(10) = e'e + cc'e , 

which depends only upon a through the sum of squares e'e. 

When conditions (5), (7), and (9) hold, the IS estimate a may be found 
by minimizing e'e , which is the error sum of squares for the model 

(11) y = g (a) + e . 

The LS problem for model (ll) is, of course, simpler than that for model 
(3) > which requires a direct minimization of 0^ in (4). The usefulness 
of (11) is illustrated in two nonlinear analyses in which (5), (7), and 
(9) are satisfied. 



Two Illustrative Analyses Involving Eckart-Young Decompositions 

Gollob's Factor Analysis of Variance (FANOVA) 

The 1 f nea r model . The linear model for the standard two-way analysis 
of variance may be written in scalar form as 



(12) y ijk = u + e . + n k + Pjk + e. jk . 

The 0. and n, are main effects, the p., 
j k jk 




are interactions, and 
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(13a) Ifl] = I\ = 0 

j k 

03b) £p. k = [p = 0 = k=1 , . . . , K) . 

j J k J 

in the J x K layout (J > K) wi th m observations per cell = the 

LS estimates are 
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u = y 
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0. = y . 

J ‘J ' 



p jk “ Y -jk '.j. ’’••k 

where a dot indicates an average over the subscript it replaces. A 
typical element of the error vector' e is 



\ = y --k ’ y . 



- v . . i. + y 



6 i j k ^ i j k ^ • j k 
and the error sum of squares is 

05) = ni(y ijk - y. jk ) 2 • 

i j k 

The nonlinear submodel, Gol lob's (1968) FANOVA model 



(16) 



y.jk p + 0j + n k + ^|^jp" pk + f jj k 



is a submodel of (12) which is formed by placing the interaction terms under 
the cons t ra i nt 



H : p . . = l X. t . 
Jk p t ] JP pk 



<j=1 J; k=1 , . . . , K) . 



The hypothesis H sets each p., equal to an inner product over r(<K) dimen- 

j K 

sions. In (16) the 6j and n k satisfy (13a), and the X Jpi and satisfy 



(17a) 



h. = It , = 0 

I JP £ pk 



jp pk 

(p=1 , . . . ,r) , 



pk 



(17b) 



(p=1 , . . . ,r) , 
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(17c) 



J A . A . = , t , -■ 0 

V jp jq £ pk qk 



( p^q ) 



Equation ( 1 7 a ) replaces ( 1 3 b ) , while (17b) and the orthogonality condi- 
tions ( 1 7 c ) are required for the uniqueness of the and under the 

factor i za t i on H . 

I dent i f i cat i ons . In examining the FANOVA model we make the following 
identifications with the preceding notation: 



6 = { u » 6 j , 



Y H {p jk } , 



e - {£j k } » 



where 



c.. = p - Ja, t . . 

jk jk p ,!P pk 

In the two-way layout with m observations per cell Z may be partitioned 

vertically, and the submatrix Z. for the i th replicate is given in Table 1 

The rows and columns are labeled by the observations y. and the inter- 

1 i j k 

actions p . ^ . Since this submatrix, which is the JK x JK identity matrix, 
is repeated for each replicate i=1,...,m , the matrix Z may be written as 






Insert Table 1 about here 
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submodel invariance of {y,0.,n, }■ The error sum of squares for ( 1 6 ) 

Jl 



<r- nif, ik • 

Ijk ,Jk 

and, introducing Lagrange multipliers <J> and ij; for (13a), we generate 

+ : + • 
j k 

Setting to zero the derivatives of ^2 with respect to y, 0j , and , 
using ( 1 3a ) and (l 7a) , and eliminating (j> and ip , we find that 



y = y , 



08 ) 



0 . = 0 . , 
J J 



\ = \ ’ 

which are given by (l4). 

The vanishing bilinear fo rm. The bilinear form in this analysis is 



e"Ze = ; 7 e. .. z. .. , e. , 

i,j,k 1 ,m 'J k ,jk ’ lm 1m 

where the element of Z i s 



1 when 1 ,m - j , k' j 



z, . 



i j k , lm ) 0 otherwi se 



Thus we have 



( 19 ) 



*' Ze = in®ni, e i 



ijk 



ijk jk 



= T Je - . I'e . .. 
jk Jk i ljk 

= 0 , 
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s i nee 
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pijk = { (y ijk ' y -jk } = 0 • 

The reduction of the quadratic form. The matrix I'l is 



z'z « y z:z. 

i-1 ' 1 



ml 



JK ‘ 



Therefore, the quadratic form may be written as 



( 20 ) 



z'Z'Zz = e^mlj^e 



= me e 



= *lb] k , 

jk 



and the error sum of squares for (16) is 



(21) 



cf ' = SZ(y,jk " y -jk )2 + • 



ijk 



jk 



jk 



The Eckart-Young decomposition of the p., . Due to (21 ) we may obtain 

JJi 

LS estimates of the X. and t . by LS fitting these parameters to the p., 

JP P k r j k 

through the model 



p = Yx . t . + e , 

jk £ jp pk jk 



which, in matrix notation, is 



( 22 ) 



P = AT + E , 



wnei e 



A A 

P = (pjjP is the J x K matrix of estimated interactions, 

A = (A. ) is a J x r matrix , 

JP 
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T = 



(x u> 

pk 



is an r * Kmatrix, 



E = 




is a J ^ K matrix of residual errors. 



The LS subproblem for (22) is easily solved by an Eckart-Young decompo- 
sition of P. This provides estimates and T which minimize 



f/z = 

jk 



2 

jk 



and hence the second term ofO?' in (21). The LS estimate of T in (22) *s 
given by 




(23) f - (i pk ) - 



\Y 

where the rows are r orthonormal (unit length) eigenvectors 

corresponding to the r(< K - 1) largest eigenvalues of the K x K, rank 
K - 1 matrix P'P (e.g., see Eckart & Young, 1936; Householder & Young, 
1938; Keller, 1962; Whittle, 1952). With T thus constructed, the LS 
estimate of A is obtained as 



(24) A = (X. ) = PT" . 

JP 

In this particular least squares solution T is row-wise orthonormal, and 
A is column-wise or thogona 1 , i.e., 

(25a) TT' = I , 

(25b) A"A = L , 

where the r diagonal elements 




£,>...>£ >0 

1 - r 

of the diagonal matrix L are the r largest eigenvalues of P"P. Equations 
(25a) and (25b) restate ( 1 7 b) and ( 1 7c) in matrix notation. 
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The Eckart-Young decompos i t i on for (22) completes the solution 



1 1 



since it provides the X and t , which, along with y . 0., and n. 5 are 

Jp P k J k 

exact LS estimates for fitting (16) to the original data y... . The 

ijk 

associated minimum error sum of squares for (16) may be found by noting that 



( 26 ) 



m!n m He.. = m£]>e 
jk jk 



K- i 

= mil, 

p=r+1 p 

where the latter summation is taken over the (K-l)-r positive eigenvalues 
not used in the construction of T (Keller, 1 962 ) . Since we have chosen the 
rows of T as r orthonormal eigenvectors corresponding to the r l argest 
eigenvalues of P^P, this summation is over the (K-l)-r smallest positive 
eigenvalues. Due to (26) the minimum of (21) is 



(27) 



= e^e + c min s'e 



ERiC 



K~1 



= Illty- jk - y. jk ) + m l t , 

ijk IJk Jk p=r+1 p 

which is the error sum of squares in LS fitting the PANOVA submodel to the 
replicated two-way layout. Expression (27) consists of the usual error 
sum of squares for this layout i ncrem en ted by a second term representing 
the (K-l)-r deleted factors in the Ecka rt-Young decomposition of the 
rank (K-l) matrix of estimated interactions. Of course, if all K 1 
factors are extracted, then the second term vanishes, and the error sum 
of squares takes a minimum value equal to that for the two-way analysis of 
va r i ance . 

The Multidimensional Choice Scaling of Bechtel, Tucker, £ Chang 

T he 1 i near model , In a three-way layout of graded paired comparisons 
a typical observation Yjj^ indicates the strength of choice for object j 
over object k in replicate i, e»g., individual i (Bechtel, Tucker & Chang, 

i :> 
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in press). When n objects are being studied, each replicate contains (^) 
comparisons, and hence there are m(^) observations in a design involving 
m (> n) replicates. The linear model for this pairwise choice layout is 



( 28 ) 



^ijk ®!j ^ik + p jk + e ijk ’ 



where the Q.. and 0.. are i n t rarep 1 i ca te scale values, the p . t are inter 
j j i k r ’ j k 



J J 

replicate interactions (Pjj. = "P^j) > and 



(29a) 

(29b) 



I® IT * ° 



u 






k*j 



( i = 1 m) 

0=1 , • • - ,n) . 



The LS estimates of the parameters of (28) are 



(30) 



°'J * ; ij y, J k ' 



' y -jk ‘ e -j + e -k • 



a typical element of the error vector e is 



e . . . = y . . , - 0 • . + 0 . , - p , 
ijk 7 i j k ij ik jk ? 

and the error sum of squares for fitting the model is 



(31) e'e = IH(y ljk ' e,j + § ik - P jk ) • 

The nonlinear submodel. The observational equation 



ERfC 



(32) 



y . . , = y x . t . - y x. t . + p + f . , 

U k ‘P PJ ^ ip Pk F jk i jk 



given by Bechtel, Tucker, and Chang (in press), is generated by placing the 
const ra i nt 



H : 6 . . = 7 X. 

II L II 



T . 



u ip PJ 



M 



( i = 1 , . . . ,m; j=1 , . . . ,n) 
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upon the scale values in (28). In (32) the p.. satisfy (29b), and the X. 

j k ip 



and t . satisfy 
PJ 



(33a) 

(33b) 

(33c) 



Ft . = 0 
j PJ 

It 2 . - 1 
j PJ 



Ft . t = £a. A. = 0 

j pj qj j ip >q 



( d= i , . , . , r ) , 
(p=1,.*.,r) , 

(p^q) 



Equation (33a) replaces (29a), while (33b) and the orthogonality conditions 

(33c) are required for the uniqueness of the A. and t . under the factor!- 

ip PJ 

zation H. 

I dentif i cati ons. The following definitions link the components of the 
choice model to the preceding general notation: 

B = (P jk > • 

yh (e. j} , 



whe re 



e = {e..} , 



e. . = 0 , . - 7 A r t 

U (J p iP PJ 



In this replicated paired comparisons design the matrix 2, which is 

m ( ^ ) x mn, may be partitioned into m^ submatrices, each being (^) x n. 

The submatrix Z..^, for replicates i and \' , is the zero matrix for 

i 7 ~ \' . For i = ]' it is the pairs x singles scale matrix in Table 2, 

where the rows and columns are labeled by the observations y... and the 

7 i j k 

scale values 8.. . Since this submatrix is invariant over replicates 
i = 1 , . . . ,m , the matrix 2 may be written as 



O 

ERLC 
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1 b 



’11 



Z = 



0 ... 0 



0 . . -Z 



. . . . 0 



^0 ... 0 ... z 



S ... 0 ... 0 



0 ... S ... 0 



V 



0 . . . s 



Insert Table 2 about here 



Submodel invarianc e of {p^ }. The error sum of squares for (32) is 



<f - n if? 



ij<k 



ijk ’ 



and, introducing Lagrange multipliers <j>j for (29b), we have 



- f * b, l P ik 
j J Wj Jk 



Setting to zero the derivative ofT^with respect to p^> using (29b) and 



(33a), and eliminating the by we find that 



( 3 *) 



p jk P jk ’ 



which is given by (30). 

The va n ishing bilinear form . The bilinear form in the error sum 

of squares for this analysis is 

e^Ze = / / e . z. . , , e, , 

. h f , L i jk i jk, Im 1m ’ 

‘ >J ,k I ,m J J 

where the element of Z i s / 

J 1 when 1 ,m = i , 

z i jk, lm = Y\ wh ^ n 1 ! m = ’’ k > * 

0 otherwi se 



erIc 



1 G 






Thus we have 



\'Zz = IJ £e...e.. - ££ Je.. b e. t 

tj<k ljk IJ ij<k ,Jk ,k 



= yy £e. .. (e. . - e. , ) 
LL L i j k i j i k 



i j <k 



and, since e.^ = -e.^, we may wr ' te 



(35) 



^ = hi I a 5! .(« 



2“ .9.""i jk' e i j e ik^ 

IJ 



2?? e 'j ii, e 1Jk ’ 2^ e ikJ k e ijk 



ij 



= 0 



Equation (35) follows from the fact that 



= Ji <v, J k ■ * e,k ' °J k> = 0 



due to (29a), (29b), (30), and the skew-symmetry of {e..^}. 

The reduction of the quadratic form . The matrix Z'2 for (28) is 



I'l = 





17 



16 



where 




Therefore, the quadratic form in the error sum of squares for (32) is 



/S'S 



'Z'Ze = 



. .S s. 




m 

7 e^S'Se. 
i-i 1 1 



where z' is the i row of the matrix (e..). Letting (l) denote an 
i U 

n x n matrix with each entry equal 1, we may reduce the i quadratic 
form under the latter summation to 



e:S"Se. = EjCnl - (Die. 



= e^nle . - zl ( 1 ) e . 
i i i i 



= ne . e . , 

i i ’ 

which follows from the fact that e."(1) is the zero vector, i.e., 



7e. . = 7(0.. - t .) = 0 

eu . i j ^ippj 

J J J P 



0*1 > . . . ,m) 



ERjt 
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due to (29a) and (33a). Using this reduction we then write the entire 
quadratic form as 

m 

( 36 ) z'Z'lz = r: 7 e'e. 

1=1 1 1 



= nil 



U 



e. . 
U 



and hence the error sum of squares for (32) as 



(37) 



ZP - n I(y Mk 

i j < k IJK 



1 . . + e M - p.. 

1 j ik jk 



) 2 + nll.f. 

ij 



The Ec kar t~Young decomposition of the 8 ,j. Equation (37) enables 

us to obtain IS estimates of the X. and t . by LS fitting these parameters 

ip PJ y 

to the 0. . through the model 
U 

S..- Tx. T . + , 

U p »P PJ u 



which, in matrix notation, is 



(38) 



0 = AT + E , 



where 

0 = (e. .) is the m x n matrix of estimated scale values, 
IJ 

A = (X. ) is an mx r matrix, 

1 p 7 

T = (x .) is an r x n matrix, 

PJ 

E = (e..) is an m x n matrix of residual errors. 

IJ 

An Eckart-Young decomposition of 0 in ( 38 ) provides estimates A and T 
which minimize 



ERIC 



= IK 



U 



and hence the second term of in (37)- The construction of A and 



19 



Hi 
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which has been described in detail by Bechtel, Tucker and Chang (in press), 

is similar to the construction of the estimated factors in the FANOVA model. 

The Ecka r t-Young decomposition for (38) completes the analysis 

by providing the X. and t . which, along with the p., , are LS estimates 
1 P PS J K 

for fitting (32) to the original paired comparisons y The error sum 
of squares for this fit is the minimum of ' in (37) and is attained 
at the minimum 



(39) 



« n-1 

nJTe. ■ = n Y £ 

• * ij £ \ 1 p 

ij J p=r+1 



of the second term of * . Since rank 0 = rank 0^6 = n-1 due to (29a) , 
the summation on the right is over the (n - 1 ) - r smallest positive eigen- 
values of 0"0. Substituting (39) into (37) gives 



m 



m i n 






e e 



c mi n e e 



- II Kv 



i jk 



e. . + e., 

i j i k 



v 2+ 



i 

i=r+ 1 



which is the error sum of squares for LS fitting the choice submodel 
to the three-way layout of paired comparisons. In (40) the error sum of 
squares for the corresponding linear model is incremented by a second 
term whenever the number r of factors in the Eckart-Young decomposition 
is less than the rank (n-l) of the matrix of estimated scale values. When 
r = n-1 the second term vanishes, and the error sum of squares is minimized 



at the first term. 
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Footnote 



The developments in this paper were initiated and carried out in 
part while the author was a Visiting Research Fellow at Educational 
Testing Service. They were extended and concluded at the Oregon Research 
Institute under Grant Nos. MH 12972 and MH 15506 from the National 
Institute of Mental Health, U. S. Public Health Service. 
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TABLE 1 

The i th submatrix Z. = I tty 

i JK 





P 1 1 • 


• • p jk • 


• • P JK 


y M1 


1 . 


. . 0 . 


. . 0 


y ijk 


0 . 


. . 1 . 


. . 0 


y i JK 


0 . 


. . 0 . 


. . 1 
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TABLE 2 



The Pairs * Singles Scale Matrix Z. . = S 

i i 





